Objective classification of residents based on their psychomotor laparoscopic skills

Background - From the clinical point of view, it is important to recognize residents’ level of expertise with regard to basic psychomotor skills. For that reason, surgeons and surgical organizations (e.g., Acreditation Council for Graduate Medical Education, ACGME) are calling for assessment tools that credential residents as technically competent. Currently, no method is universally accepted or recommended for classifying residents as ‘‘experienced,’’ ‘‘intermediates,’’ or ‘‘novices’’ according to their technical abilities. This study introduces a classification method for recognizing residents’ level of experience in laparoscopic surgery based on psychomotor laparoscopic skills alone. Methods - For this study, 10 experienced residents (>100 laparoscopic procedures performed), 10 intermediates (10– 100 procedures performed), and 11 novices (no experience) performed four tasks in a box trainer. The movements of the laparoscopic instruments were recorded with the TrEndo tracking system and analyzed using six motion analysis parameters (MAPs). The MAPs of all participants were submitted to principal component analysis (PCA), a data reduction technique. The scores of the first principal components were used to perform linear discriminant analysis (LDA), a classification method. Performance of the LDA was examined using a leave-one-out crossvalidation. Results - Of 31 participants, 23 were classified correctly with the proposed method, with 7 categorized as experienced, 7 as intermediates, and 9 as novices. Conclusions - The proposed method provides a means to classify residents objectively as experienced, intermediate, or novice surgeons according to their basic laparoscopic skills. Due to the simplicity and generalizability of the introduced classification method, it is easy to implement in existing trainers.Biomechanical EngineeringMechanical, Maritime and Materials Engineerin

Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery

Training and assessment paradigms for laparoscopic surgical skills are evolving from traditional mentor–trainee tutorship towards structured, more objective and safer programs. Accreditation of surgeons requires reaching a consensus on metrics and tasks used to assess surgeons’ psychomotor skills. Ongoing development of tracking systems and software solutions has allowed for the expansion of novel training and assessment means in laparoscopy. The current challenge is to adapt and include these systems within training programs, and to exploit their possibilities for evaluation purposes. This paper describes the state of the art in research on measuring and assessing psychomotor laparoscopic skills. It gives an overview on tracking systems as well as on metrics and advanced statistical and machine learning techniques employed for evaluation purposes. The later ones have a potential to be used as an aid in deciding on the surgical competence level, which is an important aspect when accreditation of the surgeons in particular, and patient safety in general, are considered. The prospective of these methods and tools make them complementary means for surgical assessment of motor skills, especially in the early stages of training. Successful examples such as the Fundamentals of Laparoscopic Surgery should help drive a paradigm change to structured curricula based on objective parameters. These may improve the accreditation of new surgeons, as well as optimize their already overloaded training schedules

Assessment of joystick and wrist control in hand-held articulated laparoscopic prototypes

Various steerable instruments with flexible distal tip have been developed for laparoscopic surgery. The problem of steering such instruments, however, remains a challenge, because no study investigated which control method is the most suitable. This study was designed to examine whether thumb (joystick) or wrist control method is designated for prototypes of steerable instruments by means of motion analysis. Methods: Five experts and 12 novices participated. Each participant performed a needle-driving task in three directions with two prototypes (wrist and thumb) and a conventional instrument. Novices performed the tasks in three sessions, whereas experts performed one session only. The order of performing the tasks was determined by Latin squares design. Assessment of performance was done by means of five motion analysis parameters, a newly developed matrix for assigning penalty points, and a questionnaire. Results: The thumb-controlled prototype outperformed the wrist-controlled prototype. Comparison of the results obtained in each task showed that regarding penalty points, the up ? down task was the most difficult to perform. Conclusions: The thumb control is more suitable for steerable instruments than the wrist control. To avoid uncontrolled movements and difficulties with applying forces to the tissue while keeping the tip of the instrument at the constant angle, adding a ‘‘locking’’ feature is necessary. It is advisable not to perform the needle driving task in the up down directionBiomechanical EngineeringMechanical, Maritime and Materials Engineerin

Adaptive neuro-fuzzy inference systems (ANFIS) for supervised classification of competence in MIS based on psychomotor skills

INTRODUCTION: Objective assessment of motor skills has become an important challenge in minimally invasive surgery (MIS) training.Currently, there is no gold standard defining and determining the residents' surgical competence.To aid in the decision process, we analyze the validity of a supervised classifier to determine the degree of MIS competence based on assessment of psychomotor skills METHODOLOGY: The ANFIS is trained to classify performance in a box trainer peg transfer task performed by two groups (expert/non expert). There were 42 participants included in the study: the non-expert group consisted of 16 medical students and 8 residents ( 10 MIS procedures performed) and 4 experienced surgeons. Instrument movements were captured by means of the Endoscopic Video Analysis (EVA) tracking system. Nine motion analysis parameters (MAPs) were analyzed, including time, path length, depth, average speed, average acceleration, economy of area, economy of volume, idle time and motion smoothness. Data reduction was performed by means of principal component analysis, and then used to train the ANFIS net. Performance was measured by leave one out cross validation. RESULTS: The ANFIS presented an accuracy of 80.95%, where 13 experts and 21 non-experts were correctly classified. Total root mean square error was 0.88, while the area under the classifiers' ROC curve (AUC) was measured at 0.81. DISCUSSION: We have shown the usefulness of ANFIS for classification of MIS competence in a simple box trainer exercise. The main advantage of using ANFIS resides in its continuous output, which allows fine discrimination of surgical competence. There are, however, challenges that must be taken into account when considering use of ANFIS (e.g. training time, architecture modeling). Despite this, we have shown discriminative power of ANFIS for a low-difficulty box trainer task, regardless of the individual significances between MAPs. Future studies are required to confirm the findings, inclusion of new tasks, conditions and sample population

La relevancia de las métricas de movimiento en la evaluación objetiva de habilidades motoras en cirugía laparoscópica

El presente trabajo investiga la relevancia de las métricas de movimiento en la evaluación objetiva de habilidades laparoscópicas, así como su correlación directa a dichas habilidades. Se ha realizado una validación de construcción de tres nuevas tareas para la evaluación de cirujanos. Para registrar los movimientos, se ha empleado el sistema de tracking TrEndo. 42 voluntarios (16 noveles, 22 residentes y 4 expertos) participaron en el experimento. Los resultados muestran que tiempo, camino recorrido y profundidad son métricas de evaluación válidas para un amplio espectro de habilidades. Otras métricas como la brusquedad de movimientos o el tiempo muerto demuestran validez en tareas bi-manuales. Métricas como la brusquedad o la velocidad media muestran un alto grado de independencia con respecto a los objetivos de la tarea a realizar. Se verifica la utilidad de este tipo de métricas, si bien son necesarios nuevos estudios que corroboren los resultados alcanzados

Retracting and seeking movements during laparoscopic goal-oriented movements. Is the shortest path length optimal?

Aims- Minimally invasive surgery (MIS) requires a high degree of eye–hand coordination from the surgeon. To facilitate the learning process, objective assessment systems based on analysis of the instruments’ motion are being developed. To investigate the influence of performance on motion characteristics, we examined goaloriented movements in a box trainer. In general, goal-oriented movements consist of a retracting and a seeking phase, and are, however, not performed via the shortest path length. Therefore, we hypothesized that the shortest path is not an optimal concept in MIS. Methods-Participants were divided into three groups (experts, residents, and novices). Each participant performed a number of one-hand positioning tasks in a box trainer. Movements of the instrument were recorded with the TrEndo tracking system. The movement from point A to B was divided into two phases: A-M (retracting) and M-B (seeking). Normalized path lengths (given in %) of the two phases were compared. Results- Thirty eight participants contributed. For the retracting phase, we found no significant difference between experts [median (range) %: 152 (129–178)], residents [164 (126–250)], and novices [168 (136–268)]. In the seeking phase, we find a significant difference (<0.001) between experts [180 (172–247)], residents [201 (163–287)], and novices [290 (244–469)]. Moreover, within each group, a significant difference between retracting and seeking phases was observed. Conclusions- Goal-oriented movements in MIS can be split into two phases: retracting and seeking. Novices are less effective than experts and residents in the seeking phase. Therefore, the seeking phase is characteristic of performance differences. Furthermore, the retracting phase is essential, because it improves safety by avoiding intermediate tissue contact. Therefore, the shortest path length, as presently used during the assessment of basic MIS skills, may be not a proper concept for analyzing optimal movements and, therefore, needs to be revised.Biomechanical EngineeringMechanical, Maritime and Materials Engineerin

Evaluación de habilidades motoras en cirugía de mínima invasión basada en el análisis del vídeo laparoscópico

En el presente trabajo se muestran los resultados preliminares de validación de un sistema de evaluación de habilidades motoras en cirugía de mínima invasión (CMI) basado en el seguimiento por vídeo del instrumental quirúrgico. El sistema, en fase de prototipo aún, se valida con 42 participantes (16 noveles, 22 residentes y 4 expertos). En esta primera fase del trabajo se han analizado varias métricas obtenidas a partir del 2D de la imagen. El sistema presenta validez para las métricas de tiempo, camino recorrido, velocidad media y eficiencia de espacio. A falta de implementar una versión definitiva capaz de llevar a cabo procesamiento en tiempo real de múltiples herramientas, y de la validación de métricas obtenidas a partir del 3D, se demuestra la viabilidad de construir entornos de formación basados en tecnologías de video, que sean transparentes al cirujano

Relevance of motion-related assessment metrics in laparoscopic surgery

INTRODUCTION: Motion metrics have become an important source of information when addressing the assessment of surgical expertise. However, their direct relationship with the different surgical skills has not been fully explored. The purpose of this study is to investigate the relevance of motion-related metrics in the evaluation processes of basic psychomotor laparoscopic skills, as well as their correlation with the different abilities sought to measure. METHODS: A framework for task definition and metric analysis is proposed. An explorative survey was first conducted with a board of experts to identify metrics to assess basic psychomotor skills. Based on the output of that survey, three novel tasks for surgical assessment were designed. Face and construct validation study was performed, with focus on motion-related metrics. Tasks were performed by 42 participants (16 novices, 22 residents and 4 experts). Movements of the laparoscopic instruments were registered with the TrEndo tracking system and analyzed. RESULTS: Time, path length and depth showed construct validity for all three tasks. Motion smoothness and idle time also showed validity for tasks involving bi-manual coordination and tasks requiring a more tactical approach respectively. Additionally, motion smoothness and average speed showed a high internal consistency, proving them to be the most task-independent of all the metrics analyzed. CONCLUSION: Motion metrics are complementary and valid for assessing basic psychomotor skills, and their relevance depends on the skill being evaluated. A larger clinical implementation, combined with quality performance information, will give more insight on the relevance of the results shown in this study

EVA: Laparoscopic instrument tracking based on endoscopic video analysis for psychomotor skills assessment

INTRODUCTION: The EVA (Endoscopic Video Analysis) tracking system a new tracking system for extracting motions of laparoscopic instruments based on non-obtrusive video tracking was developed. The feasibility of using EVA in laparoscopic settings has been tested in a box trainer setup. METHODS: EVA makes use of an algorithm that employs information of the laparoscopic instrument's shaft edges in the image, the instrument's insertion point, and the camera's optical centre to track the 3D position of the instrument tip. A validation study of EVA comprised a comparison of the measurements achieved with EVA and the TrEndo tracking system. To this end, 42 participants (16 novices, 22 residents, and 4 experts) were asked to perform a peg transfer task in a box trainer. Ten motion-based metrics were used to assess their performance. RESULTS: Construct validation of the EVA has been obtained for seven motion-based metrics. Concurrent validation revealed that there is a strong correlation between the results obtained by EVA and the TrEndo for metrics such as path length (p=0,97), average speed (p=0,94) or economy of volume (p=0,85), proving the viability of EVA. CONCLUSIONS: EVA has been successfully used in the training setup showing potential of endoscopic video analysis to assess laparoscopic psychomotor skills. The results encourage further implementation of video tracking in training setups and in image guided surgery